Introduction

The PISA 2018 results were released on 3 December 2019. This led to wringing of hands in the Australian press, with titles of stories like Vital Signs: Australia’s slipping student scores will lead to greater income inequality and In China, Nicholas studied maths 20 hours a week. In Australia, it’s three. Australia’s neighbours, New Zealand and Indonesia, are also worrying: New Zealand top-end in OECD’s latest PISA report but drop in achievements ‘worrying’, Not even mediocre? Indonesian students score low in math, reading, science: PISA report.

The data from this survey and all of the surveys conducted since the first collection in 2000, is publicly available. We have made a more convenient subset of the data available in a new R package, called learningtower, along with sample code for analysis, openly available. What you can learn by looking at the data, is that the variation from individual is vastly dominates any difference between averages. Focusing attention on the averages, can be convenient, from a public policy perspective, it detracts from paying attenton to the individual, and boosting opportunities for all.

About the data

Since 2000, the OECD has overseen a Programme for International Student Assessment (PISA). The goal of this programme it to provide reliable indicators of what people actually know and can do. Details can be found at https://www.oecd.org/pisa/. Every three years 15 year old students around the world are tested on standardised skills in mathematics, science and reading. The students are also surveyed on aspects of their everyday life, that includes resources, quality of life and study habits. In addition, surveys are conducted with the school administrators and parents, about resources and quality of life.

The scope is gradually broadening, with more countries being added each three years. 2018 had results from 80 countries, double the initial number of 43. Each country is responsible for their data collection, under OECD oversight, and this is mostly consistent with a few notable exceptions, like China and the USA. In Australia, for the 2018 survey, ??? schools were selected for testing of 14273 students. Stratified sampling was conducted to ensure representation from each state and territory, each type of school (public, private and catholic). A balance across urbanisation, socieoeconomic status, gender was checked. An average of 20 students are tested from each school.

China samples schools and students selectively, with OECD approval, and its not entirely clear that the data is representative of the whole country. (See a Washington Post article by Valerie Strauss for commentary on China’s PISA practices.) In the USA, the procedures for data collection are carefully documented and approved by the OECD, but the sample of schools and students is relatively small, about a third the numbers sampled in Australia. Schools can refuse to take part. (See the technical notes for the USA PISA surveys for details.)

The data from the testing, and the responses to the survey questions from students, parents and schools is freely available for download. It needs to be noted, though, that the raw test scores are not released. You can see this f you have a keen eye. The plots below show the 2018 scores for Australian students: a histogram of the math scores is plotted at left, and a scatterplot comparing math and reading scores is plotted at right. The histogram is beautifully bell-shaped and the scatterplot has a lovely elliptical appearance. These perfect structures should inspire suspician, and indeed if you read the data documentation you learn that the released values are simulated from a complex linear model applied to the data. Actually, multiple values are simulated for each student. This is called synthetic data, which is a common approach to ensure data privacy, and the data can still be considered to be accurate within the mean, variance and strata (country, gender, …) used in the model of the original data.

Is it really this dramatic?

Richard Holden’s Dec 6, 2019 article in the Conversation shows the figure below, along with an alarming statement “Australian 15-year-olds are now below the OECD average in mathematics, and our results in reading and science have fallen badly.”

A reason that this looks so dramatic, is that the scale of the differences is magnified. The repoorted scores range from 0 through 1000. These plots zoom in to the neighbourhood of 500. The animation below shows the average math scores on the larger scale of the data. For comparison several other countries are shown: Canada, Finland, Great Britain, Japan, New Zealand, Singapore and the USA.

There has been a decline of 25 points in Australian students math scores on average, since 2000, which sounds bad, and particularly if the plot is made on a range of 450-550 also looks terrible. (It should be noted that the average for 2018 is similar to that for 2015.) If these values are examined on the full data scale, the line looks flat, and particularly is not very different from related countries. All years the average is higher than the USA. Except for this past year, the averages were all higher than those in Great Britain. New Zealand averages are similar to Australia. Finland, the education shining star, has also seen a decline in average math score, but in contrast to Australian news headlines in Finland the title is Finland Remains Among Top Nations in PISA Education Survey.

All of the scores fluctuate, and it is arguable that the level of fluctuation is within sampling and modeling error.

Exploring PISA

Individuals

In the plot below, one dot represents one student. The scores for students in each country are very spread out: the lowest scores for each country are similar, and the highest scores for each country are similar. Even, though Singapore has a higher median score, the top math score in Australia is higher than the top math score in Singapore, and Canada has the highest individual overall math score.

Gender gap

Plot of gender gap over time, in math and reading.

Something’s off with gender - cross-checking against the PISA Australia summary, and it doesn’t look correct here.

Do it yourself

The PISA data across all years is provided in a convenient format in the R package, learningtower available from https://github.com/ropenscilabs/learningtower. Details on how to install the package, and access the data are at https://ropenscilabs.github.io/learningtower/.

The work to make the data available is the effort of several researchers from Australia, New Zealand and Indonesia, conducted as part of the ROpenSci OzUnconf held in Sydney, Dec 11-13, 2019.